Papers with Gemini 2.5 Pro
Literary Evidence Retrieval via Long-Context Language Models (2025.acl-short)
Copied to clipboard
| Challenge: | a recent study shows that long-context language models can exceed human expert performance in literary analysis . despite their speed and apparent accuracy, even the strongest models struggle with nuanced literary signals and overgeneration. |
| Approach: | They propose a task where a model is given an entire text of a book and a literary criticism with a missing quotation from that work and asked to generate the missing quote. |
| Outcome: | The proposed model outperforms open-weight models in literary evidence retrieval tasks. |
Agent vs. Agent: Automated Data Generation and Red-Teaming for Custom Agentic Workflows (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Existing red-teaming frameworks like AgentHarm use static prompts and hardcoded toolsets . |
| Approach: | They propose a red-teaming framework that generates adversarial tasks and evaluation functions tailored to arbitrary toolsets and uses iterative prompt refinement with self-reflection to develop more effective attacks. |
| Outcome: | The proposed approach achieves 162% increase in attack success rate on o4-mini and 86% success on gemini 2.5 Pro. |
Tonal Salience in Cognitive Decline: In-Context MCI Detection with Multimodal LLMs (2026.acl-srw)
Copied to clipboard
| Challenge: | a recent study shows that tonal languages like Chinese have a higher classification performance than non-tonal languages like English. |
| Approach: | a new study examines the differences between tonal and non-tonal language classifications . they hypothesize that the difference is rooted in language typology . early cognitive decline is notoriously difficult to detect . |
| Outcome: | The proposed method compared to TAUKADIAL audio shows that Chinese and English perform better on Chinese . the findings suggest that language typology should inform the design of audio-based cognitive screening tools . |
What Users Leave Unsaid: Under-Specified Queries Limit Vision-Language Models (2026.findings-acl)
Copied to clipboard
Dasol Choi, Guijin Son, Hanwool Lee, Minhyuk Kim, Hyunwoo Ko, Teabin Lim, Eungyeol Ahn, Jungwhan Kim, Seunghyeok Hong, Youngsook Song
| Challenge: | HAERAE-Vision benchmarks feature clear, explicit prompts but are often informal and underspecified . state-of-the-art models achieve under 50% on original queries, compared to GPT-5 and Gemini 2.5 Pro . |
| Approach: | They propose a benchmark of 653 real-world visual questions from Korean online communities . they find that even state-of-the-art models achieve under 50% on original queries . |
| Outcome: | HAERAE-Vision benchmarks from Korean online communities yield 1,306 query variants . state-of-the-art models achieve under 50% on original queries, compared with smaller models . authors show that query explicitation alone yields 8 to 22 point improvements . |
Protecting Bystander Privacy via Selective Hearing in Audio LLMs (2026.acl-long)
Copied to clipboard
| Challenge: | Audio Large language models capture speech from unintended bystanders, raising privacy risks that existing benchmarks and defences did not consider. |
| Approach: | They propose to evaluate selective hearing by evaluating a model’s ability to attend to an intended main speaker while refusing to process or reveal information about incidental bystander speech. |
| Outcome: | The proposed model can attend to an intended main speaker while refusing to process or reveal information about incidental bystander speech. |
CHURRO: Making History Readable with an Open-Weight Large Vision-Language Model for High-Accuracy, Low-Cost Historical Text Recognition (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing vision-language models are not equipped to read diverse languages and scripts found in historical materials. |
| Approach: | They propose to train an open-weight vision-language model for historical text recognition on CHURRO-DS, the largest historical text-recognition dataset to date. |
| Outcome: | The proposed model outperforms existing vision-language models on CHURRO-DS, the largest historical text recognition dataset to date. |
SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants? (2025.emnlp-main)
Copied to clipboard
Yao Dou, Michel Galley, Baolin Peng, Chris Kedzie, Weixin Cai, Alan Ritter, Chris Quirk, Wei Xu, Jianfeng Gao
| Challenge: | Large language models (LLMs) are increasingly used in interactive applications, and human evaluation remains the gold standard for assessing their performance in multi-turn conversations. |
| Approach: | They propose to use large language models to simulate users for automatic assistant evaluation. |
| Outcome: | The proposed model outperforms human evaluations on two interactive tasks and achieves Spearman’s of 0.7 on both tasks. |